Information processing apparatus, information processing method, and storage medium

ABSTRACT

To measure a three-dimensional shape of a measurement target object at high speeds, an information processing apparatus is configured to acquire an image captured while a pattern including first features and a second feature is projected onto the measurement target object; identify the first features in the captured image, and acquire an initial value of shape information on the measurement target object in an area including the first features; set a candidate for the shape information in a first area based on an initial value of shape information in an area surrounding the first area; decide an area in the pattern corresponding to the first area as a second area, and evaluate a correlation between the second area and the first area based on the second feature in the first area and the second feature in the second area; and derive the shape information in the first area.

BACKGROUND OF THE INVENTION

Field of the Invention

The present disclosure generally relates to information processing and,more particularly, to an information processing apparatus, aninformation processing method, a storage medium, and a technique formeasuring a three-dimensional shape of a measurement target objectsurface based on an image of the measurement target object captured witha pattern projected onto the target object.

Description of the Related Art

There is a conventional measurement method by projecting an illuminationpattern including a design onto a target object and measuring a distancebased on an image of the observed target object. According to thismethod, a correspondence between the image and the illumination patternis calculated and the distance is determined on the principle oftriangulation method. In the calculation of the correspondence,positions on the illumination pattern are changed to search forpositions matching to positions on the image. In general, the searchprocess takes time depending on the search area.

The specification of U.S. Patent Application Publication No.2010/0118123 discloses a method using an illumination pattern with dotsas design elements.

In the case of using the pattern described in the specification of U.S.Patent Application Publication No. 2010/0118123, it takes time to searchfor the correspondence between the image and the illumination pattern.

In view of these issues, aspects of the present disclosure allowhigh-speed measurement of a three-dimensional shape of an object.

SUMMARY OF THE INVENTION

According to an aspect of the present disclosure, an informationprocessing apparatus includes: an acquisition unit that acquires animage captured by an imaging apparatus while pattern including aplurality of lines, identification information disposed on or betweenthe lines for identification of the lines, and features disposed betweenthe plurality of lines is projected onto a measurement target object; aninitial value acquisition unit that associates the lines included in theprojected pattern with the lines included in a captured image based onthe identification information included in the captured image toidentify the lines included in the captured image, and acquires aninitial value of shape information on the measurement target object inan area including the lines in the captured image based on identifiedresults.

A setting unit that sets a candidate for the shape information in afirst area within the captured image based on the initial value of theshape information in an area surrounding the first area; an evaluationunit that decides an area in the pattern corresponding to the first areaas a second area based on the candidate for the shape information in thefirst area, and evaluates a correlation between the second area in thepattern corresponding to the first area and the first area based on thefeatures included in the first area and the features included in thesecond area; and a derivation unit that derives the shape informationfor the first area based on the correlation evaluated by the evaluationunit.

According to the present disclosure, it is possible to measure thethree-dimensional shape of an object at high speeds.

Further features of the present disclosure will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an illumination pattern acquired by aninformation processing apparatus in a first embodiment.

FIG. 2 is a diagram illustrating a configuration of the informationprocessing apparatus in the first embodiment.

FIG. 3 is a flowchart of information processing in the first embodiment.

FIG. 4 is a diagram illustrating an example of an image in the firstembodiment.

FIG. 5 is a diagram illustrating variations of reference features in asecond modification example.

FIG. 6 is a diagram illustrating variations of designs in a thirdmodification example.

FIG. 7 is a diagram illustrating a hardware configuration of aninformation processing apparatus of the present disclosure.

DESCRIPTION OF THE EMBODIMENTS

Various exemplary embodiments of the present disclosure will bedescribed in detail below with reference to the attached drawings.

Prior to description of embodiments according to the present disclosure,a hardware configuration of an information processing apparatus in eachof the embodiments will be explained with reference to FIG. 7.

FIG. 7 is a diagram of a hardware configuration of an informationapparatus in each of the embodiments. Referring to FIG. 7, a centralprocessing unit (CPU) 710, which may include one or more processors andone or more memories, performs a centralized control of devicesconnected together via a bus 700. The CPU 710 reads processing steps andprograms from a read-only memory (ROM) 720 and executes the same. Anoperating system (OS), processing programs, device drivers, and othersaccording to each of the embodiments are stored in the ROM 720, and aretemporarily stored in a random access memory (RAM) 730 and are executedas appropriate by the CPU 710. An input. I/F 740 inputs signals fromexternal devices (display devices, operating devices, and others) in aform capable of being processed by the information processing apparatus1. An output I/F 750 outputs signals to external devices (displaydevices) in a form capable of being processed by the display devices. Asused herein, the term “unit” generally refers to any combination ofsoftware, firmware, hardware, or other component, such as circuitry,that is used to effectuate a purpose.

First Embodiment

In relation to a first embodiment, a method for measuring shapeinformation on a measurement target object at high speeds will beexplained.

In the embodiment, the shape information indicates the values ofdistances to individual parts of the measurement target object withreference to a coordinate system of an imaging apparatus. As illustratedin FIG. 1, an illumination pattern 100 for use in the embodiment iscomposed of one or more lines 101 (hereinafter, the lines will be calledreference features) and a design with dots 102 as elements. Thereference features have identification information 103 for identifyingthe reference features on the illumination pattern 100 to whichreference features observed on the image belong. In the embodiment, theidentification information is represented by a plurality of cuts in thelines.

In the embodiment, first, positions on the illumination pattern 100corresponding to the reference features on an image of the observedmeasurement target object are calculated to measure shape information onthe measurement target object onto which the reference features areprojected. Then, as for respective parts on the image, candidates forthe shape information are set based on the shape information measuredfrom the reference features. Next, as for the candidates for the shapeinformation set for the image parts, the degrees of correspondencebetween the image parts and corresponding parts on the illuminationpattern 100 equivalent to the candidates for the shape information arecalculated based on the correlations between image areas including theimage parts and areas on the illumination pattern 100 including thecorresponding parts. In this example, the degrees of correspondenceindicate the heights of correlations between the image areas includingthe noted image parts and the areas on the illumination pattern 100. Thedegrees of correspondence are calculated by block matching or the like,for example. Finally, out of the candidates for the shape informationfor which the degrees of correspondence with the image parts have beencalculated, the candidates for the shape information with high degreesof correspondence are output as correct shape information for the imageparts.

In the embodiment, the candidates for the shape information are set forthe image parts from the shape information calculated from the referencefeature. The solution is selected from among the set candidates, whichdecreases the amount of calculation as compared to the calculation withall the shape information within the search area as candidates. Thisallows high-speed measurement of the shape information on themeasurement target object.

FIG. 2 illustrates a configuration of the embodiment. First, aninformation processing apparatus 200 of the embodiment is composed of apattern acquisition unit 210, an image acquisition unit 220, a referenceshape information estimation unit 230, a candidate setting unit 240, acandidate evaluation unit 250, and a candidate selection unit 260.Reference sign 300 indicates a projection apparatus, and 400 an imagingapparatus. Reference sign 500 indicates a measurement target object.

The projection apparatus 300 is a projector that projects an image ofthe illumination pattern. 100 by light or the like. Intrinsic parameterinformation on the projection apparatus 300 such as a focal length, aprincipal point position, and a lens distortion parameter, andinformation on position and orientation of the projection apparatus 300relative to the imaging apparatus 400 are calibrated in advance. Thecalibration can be performed by a publicly known method, for example,the one described in the following document:

-   M, Kimura, “Projector Calibration using Arbitrary Planes and    Calibrated Camera “Computer Vision and Pattern Recognition, CVPR,    2007.

The imaging apparatus 400 is a camera capturing grayscale images.Intrinsic parameter information on the imaging apparatus 400 such as afocal length, a principle point, and a lens distortion parameter arecalibrated in advance. The calibration can be performed by a publiclyknown method, for example, the one described in the following document:

-   R. Y, Tsai, “A versatile camera calibration technique for    high-accuracy 3D machine vision metrology using off-the-shelf TV    camer and lenses” IEEE Journal of Robotics and Automation, vol.    RA-3, no. 4, 1987.

The pattern acquisition unit 210 acquires an image of the illuminationpattern 100 for projection onto the measurement target object 500. Theacquired Pattern is a two-dimensional image that is composed of one ormore reference features (first features) and a design (second feature)as illustrated in FIG. 1. In the embodiment, the reference features arelines and the design is dots. The dots are arranged at random with apredetermined capability (A%), for example. The identificationinformation (cuts in the lines) is arranged on the lines such that aspace between the adjacent cuts in the same line becomes B+ΔB. In thisexample, ΔB indicates a uniform random number occurring within a rangeof “−ε to ε” where ε denotes a fixed value specified by the user. FIG. 1illustrates the illumination pattern 100 in which the background is inwhite and the reference features and the design are in black. The lightof the pattern is reversed such that the background is projected withlow brightness and the reference features and the design are projectedwith high brightness.

The image acquisition unit 220 acquires an image 700 (captured image)obtained by the imaging apparatus 400 imaging the measurement targetobject 500 with projection of the illumination pattern 100. FIG. 4illustrates an example of the acquired image 700.

The reference shape information estimation unit 230 calculates positionson the illumination pattern 100 corresponding to the reference featuresobserved on the image 700 based on the reference signs included in thereference features to measure the shape information on the measurementtarget object onto which the reference features are projected. The shapeinformation in the embodiment refers to the values of distances to theparts of the measurement target object with reference to the coordinatesystem of the imaging apparatus.

The candidate setting unit 240 sets the candidates for shape informationin image parts on the image 700 based on the shape information of thereference features on the image 700 estimated by the reference shapeinformation estimation unit 230.

In the embodiment, the image parts on the image 700 refer to pixels.

For the candidates for the shape information set on the image parts onthe image 700, the candidate evaluation unit 250 evaluates the degreesof correspondence between image areas including the image parts andareas including the corresponding parts on the illumination pattern 100equivalent to the candidates for the shape information, based on thecorrelation between the areas.

For the image parts on the image 700, the candidate selection unit 260selects, from among the candidates for which the candidate evaluationunit 250 have calculated the degrees of correspondences, the candidateswith high degrees of correspondence as correct shape information in theimage parts.

These functional units are implemented by the CPU 710 developing theprograms stored in the ROM 720 on the RAM 730, and executing processesaccording to flowcharts described later. Alternatively, when hardware isprovided as a substitute for software processing using the CPU 710, forexample, an arithmetic units or circuit may be configured incorrespondence with the processes of the functional units describedherein.

Next, a processing procedure of the embodiment will be explained. FIG. 3is a flowchart of the processing procedure of the embodiment.

(S610)

At S610, the pattern acquisition unit 210 acquires an image of theillumination pattern 100. Then, the pattern acquisition unit 210 sendsthe same to the projection apparatus 300 via an interface notillustrated. The projection apparatus 300 acquires the pattern from thepattern acquisition unit 210 and projects the acquired pattern onto aspace including the measurement target object. 500. While the projectionapparatus 300 projects the acquired pattern onto the space including themeasurement target object 500, the imaging apparatus 400 captures animage of the space.

(S620)

At S620, the image acquisition unit 220 acquires the image 700 capturedby the imaging apparatus 400 from the image of the illumination pattern100 projected onto the measurement target object 500. FIG. 4Aillustrates an example of the acquired image 700.

(S630)

At S630, the reference shape information estimation unit 230 calculatesthe positions on the illumination pattern 100 corresponding to thereference features observed on the image 700 to estimate the shapeinformation on the measurement target object (acquisition of the shapeinformation).

First, the reference shape information estimation unit 230 detects thereference features from the image 700. In the embodiment, the referencefeatures are lines, and therefore the lines are detected. Specifically,the reference shape information estimation unit 230 applies a Sobelfilter to the image 700 to apply extreme values of brightness on theimage (hereinafter, called detected points). Next, the reference shapeinformation estimation unit 230 labels adjacent detected points todetect the reference features as the lines in which the adjacentdetected points are aligned.

Next, the reference shape information estimation unit 230 associates thedetected reference features with one another based on the identificationinformation 103 (the cuts in the lines) included in the referencefeatures. The association of the reference features here refers tocalculating the positions of the reference features on the illuminationpattern 100 corresponding to the reference features detected from theimage 700.

The method for association with the use of the cuts is publicly knownand described in Japanese Patent Application Laid-Open No. 1-274007.Specifically, first, for the reference features detected from the image700, two cuts in the same reference feature are selected, and twoepipolar lines are calculated on the illumination pattern 100. Thepositions on the illumination pattern 100 corresponding to the cuts inthe reference feature detected from the image 700 are estimated to becuts as geometric features existing on the epipolar lines calculated onthe illumination pattern 100. According to this method, the positionscorresponding to the detected reference features are calculated bysearching for the reference features in which there is a match betweenthe two epipolar lines calculated from the cuts and the cuts in thereference features on the illumination pattern 100. Accordingly, it ispossible to calculate the positions on the illumination pattern 100corresponding to the reference features detected from the image 700.

Next, based on the results of the association of the reference features,the reference shape information estimation unit 230 estimates the shapeinformation in the reference features observed on the image 700. Theshape information (the values of distances in the embodiment) iscalculated from the positions of the reference features on the image 700and the corresponding positions on the illumination pattern 100 based onthe principles of triangulation method. In the embodiment, the referencefeatures are lines and therefore the method for calculation is the sameas the method for calculation of the shape information based on theoptical cutting method.

Hereinafter, out of the reference features detected from the image 700,parts on the reference features on which the shape information has beenestimated will be designated as reference parts Fi (i=1 to M), and theirshape information as Si. In this example, the suffix i denotes thenumber for each of the parts on the reference features, and M denotesthe total number of the parts on which the shape information has beenestimated. The reference parts Fi have parameters indicative oftwo-dimensional coordinates on the image 700. The shape information Sihas one-dimensional distance values (Di) in the embodiment. Accordingly,the reference shape information estimation unit 230 estimates the shapeinformation (acquires the initial values of the shape information) inthe areas including the reference features (lines) within the capturedimage.

(S640)

The candidate setting unit 240 sets the candidates for the shapeinformation in the respective image parts Pk (k=1 to N) on the image 700(within the captured image) based on the shape information Si on thereference parts Fi estimated by the reference shape informationestimation unit 230.

In the embodiment, the reference sign Pk denotes all the pixels on theimage 700. The suffix k denotes the numbers for the pixels, and thesuffix N the total number of the pixels. The reference parts Fi haveparameters indicative of two-dimensional coordinates on the image 700.

In this example, the candidate setting unit 240 propagates the shapeinformation Si possessed by the reference parts Fi to the surroundingimage parts Pk, and set the same as candidates for the shape informationin the image parts Pk. Specific process details will be described below.

First, the candidate setting unit. 240 selects the reference parts Fi asreferences for setting the candidates. In the embodiment, 0 to M aresequentially selected with changes in the number i for the referenceparts.

Next, the candidate setting unit 240 sets the candidates for the shapeinformation in the image parts Pk surrounding the reference parts(reference areas) Fi, based on the shape information Si on the referenceparts Fi. Specifically, when the candidate for the shape information inthe image part Pk existing within a predetermined range R from thereference parts Fi is designated as Ski, the candidate setting unit 240sets Ski =Si. When there is a plurality of image parts Pk within thepredetermined range R, the candidate setting unit 240 sets Si as thecandidate for the shape information in each of the image parts.

The candidate setting unit 240 performs the foregoing process on one ormore reference parts Fi. In the embodiment, the candidate setting unit240 performs this calculation on all the reference parts Fi (i =1 to M).Since the candidates are set with reference to the plurality ofreference parts Fi, one image part may have a plurality of candidates.

(S650)

The candidate evaluation unit 250 calculates the degrees ofcorrespondence between the candidates for the shape information set onthe image parts Pk (first areas) by the candidate setting unit 240,based on the correlation between the areas on the image 700 includingthe image parts Pk and the areas including the corresponding parts onthe illumination pattern 100 equivalent to the shape information. Theimage areas of the image parts Pk on the image include the referencefeatures and the design projected by the projection apparatus 300.Therefore, calculating the correlation between the areas meanscalculating whether there is a match between the reference features andthe design in the areas.

First, the candidate evaluation unit 250 decides corresponding parts Qkion the illumination pattern 100 (within the pattern) corresponding tothe image parts Pk on which the candidates for the shape information areset, based on the candidates Ski for the shape information in the imageparts Pk. In this example, the corresponding parts Qki have parametersindicative of two-dimensional coordinates on the illumination pattern100. The corresponding parts Qki can be determined by calculatingthree-dimensional positions in the coordinate system of the imagingapparatus 400 from the values of distances included in the shapeinformation and then projecting the calculated three-dimensionalpositions onto the illumination pattern. 100 based on the relativepositions and orientations of the imaging apparatus 400 and theprojection apparatus 300.

Next, the candidate evaluation unit 250 calculates the degrees ofcorrespondence by block matching based on the areas surrounding thereference parts Pk and the areas surrounding the corresponding parts Qkion the pattern. In this example, the areas surrounding Pk and Qki arerectangular blocks of size W centered on the respective positions. Thedegree of correspondence is set as the reciprocal number of SSD (Sum ofSquared Difference). That is, as the value of the SSD is smaller (i.e.,the difference between the blocks is smaller), the degree ofcorrespondence becomes larger.

(S660)

The candidate selection unit 260 selects, from among the candidates forwhich the degrees of correspondence have been calculated in the imageparts Pk on the image, the candidates with high degrees ofcorrespondence as correct shape information in the image parts Pk. Inthe embodiment, the candidate selection unit 260 selects the candidateswith degrees of correspondence equal to or higher than a predeterminedthreshold value T. When there is a plurality of candidates for one part,the candidate selection unit 260 selects the candidate with the highestdegree of correspondence from among the candidates with the degrees ofcorrespondence equal to c higher than the threshold value T.

As described above, in the embodiment, the candidates for the shapeinformation are set in the image parts from the shape informationcalculated from the reference features. Since the solution is selectedfrom. among the set candidates, the amount of calculation becomessmaller as compared to the case in which the calculation is performedwith all the shape information within the search area as candidates.This makes it possible to measure (derive) the shape information on themeasurement target object at high speeds.

In the embodiment, the projection apparatus 300 may be any device thatprojects a two-dimensional illumination pattern. For example, theprojection apparatus 300 may be a document projector that projectstwo-dimensional images or may be a device with a combination of a lightsource and a mask pattern. The pattern to be projected may be agrayscale pattern or a color pattern. In the case of setting a maskpattern, the pattern is physically set in the projection apparatus 300,and therefore it is not necessary to input the pattern from the patternacquisition unit 210.

In the foregoing explanation, the reference features and the design onthe illumination pattern are projected with higher brightness than thebackground on the illumination pattern. Alternatively, the referencefeatures and the design on the illumination pattern may be any featuresthat are different in brightness or colors as compared to the backgroundon the illumination pattern. For example, the reference features and thedesign on the illumination pattern may be projected with lowerbrightness than the background on the pattern. Pattern variations willbe described below in relation to a fourth modification example.

The pattern may be acquired from a storage area not illustrated in theinformation processing apparatus 200 or from an external informationsource via a network. The storage area may be a hard disc or a RAM.

The image acquired by the image acquisition unit 220 may be anytwo-dimensional image captured by the imaging apparatus while theillumination pattern is projected onto the measurement target object.For example, the image may be a grayscale image or a color image. As themethod for acquiring the image, the image may be acquired directly fromthe imaging apparatus 400 or the image stored temporarily in the memorymay be acquired later.

The image parts in which the shape information is estimated may bepixels of the image as in the first embodiment or may be a plurality ofblocks divided from the image. The parts may be identical or differentin size or shape.

In the foregoing explanation, the reference shape information estimationunit. 230 outputs the shape information only in point groups on thereference features. Alternatively, the shape information for pointgroups between the reference features may be calculated by interpolationor the like, and be output as dense shape information. The interpolationmay be performed by any method for calculating the shape informationbetween the reference features based on the shape information in thereference features. For example, the image may be divided into aplurality of areas, and the shape information existing in the areas maybe interpolated based on average values of shape information existing inthe areas Alternatively, out of the point groups on the referencefeatures for which the shape information has been estimated on theimage, three adjacent points may be connected to form a triangular patchand then the pixels in the patch may be interpolated from the shapeinformation in the three points. When the reference features are lines,the shape information between two adjacent lines may be set by linearinterpolation. The output shape information may be all or some of thecalculated point groups For example, only the shape information in thepoint groups within a predetermined range of the image may be output, oronly the shape information in the point groups within a range of apredetermined distance may be output. The shape information determinedby the interpolation may be used as reference parts for the candidatesetting unit 240 to set candidates.

In the embodiment, the candidate setting unit 240, the candidateevaluation unit 250, and the candidate selection unit 260 may repeat theprocesses at S640, S650, and S660. In this case, the candidate settingunit 240 sets the candidates for the shape information again based onthe shape information in the image parts selected by the candidateselection unit 260. Specifically, after the completion of the process bythe candidate selection unit 260 in the first cycle, the candidatesetting unit 240, the candidate evaluation unit 250, and the candidateselection unit 260 repeat the processes in this order. In the second andsubsequent cycles, the candidate setting unit 240 sets the candidatesfor the shape information with reference to the shape information in theimage parts selected by the candidate selection unit 260 in the previouscycle, instead of the shape information in the reference parts. Thenumber of repetitions may be set under any standard. For example, theprocesses may be repeated a predetermined number of times or may berepeated until the degrees of correspondence between the shapeinformation selected by the candidate selection unit 260 reach apredetermined value or less.

The candidate setting unit 240 sets the candidates for the shapeinformation with reference to the one or more reference parts Fi. Theparts to be referenced may be all the reference parts Fi as in the firstembodiment or may be some of the reference parts Fi. When the candidatesetting unit 240, the candidate evaluation unit 250, and the candidateselection unit 260 repeat the processes at S640, S650, and S660, thereference parts may be changed at each repetition. For example, thereference part may be selected with changes in the number i for thereference parts such that the reference part Fi with i=1 is used in thefirst cycle, and the reference part Fi with i=2 is used in the secondcycle.

Second Embodiment

In a second embodiment, a method for treating the distance values of themeasurement target object and the orientations of the planes of themeasurement target object (normal information) as the shape informationwill be explained. The normal information here indicates the amount ofchange in the distance value relative to changes in the positions of thepixels on the image 700. Specifically, the variation of the distancevale relative to an x axis (lateral direction) of the image isdesignated as A, and the variation in the distance value relative to a yaxis (vertical direction) of the image is designated as B. The twoparameters indicate the normal information. The distance value isdesignated as D.

In the embodiment, the reference shape information estimation unit 230,the candidate setting unit. 240, and the candidate evaluation unit 250perform different processes from those in the first embodiment.

First, the reference shape information estimation unit 230 estimates thedistance values and the normal information in the reference featuresobserved on the image 700. The distance values are estimated by themethod described above in relation to the first embodiment. The normalinformation is estimated based on spatial changes in the estimateddistance values. Specifically, for example, the distance values of thepixels between the reference features are interpolated, and then thedifferences between the distance values of adjacent pixels are taken toset values Ai and Bi. As described above, the reference shapeinformation estimation unit 230 of the embodiment estimatesthree-dimensional data (Di, Ai, and Bi) as the shape information Si onthe reference parts Fi.

Next, the candidate setting unit 240 sets the candidates for the shapeinformation in the image parts Pk on the image, based on the distancevalues and the normal information estimated from the reference features.The candidate for the shape information set in the image part Pkexisting within a predetermined range R from the reference part Fi isdesignated as Ski=(Dki, Aki, and Bki). The differences between thepositions of the reference part Fi and the image part Pk on the image700 are designated as Uki and Vki. In this example, the reference signUki indicates the difference in the x-axis direction of the image, andthe reference sign Vki indicates the difference in the y-axis directionof the image. The distance value Dki is set as Dki=Di +Ai·Uki+Bi·Vkitaking into account changes in the distance value in the normalinformation. The normal information Aki and Bki is set as Aki=Ai andBki=Bi. A in the first embodiment, when there is a plurality of imageparts Pk within the predetermined range R, the candidate Ski for theshape information is set for each of the image parts Pk. However, thecandidates are set taking into account changes in the distance value inthe normal information as described above.

Next, the candidate evaluation unit 250 calculates the degrees ofcorrespondence between the candidates for the shape information set inthe image parts Pk by the candidate setting unit 240, based on thecorrelation between the areas on the image including the image parts Pkand the areas including the corresponding parts on the illuminationpattern 100 equivalent to the shape information. In the embodiment, thecorrelation between the blocks is calculated as in the first embodiment.In the embodiment, however, the blocks are deformed and projected ontothe illumination pattern 100 based on the gradients of the planes of themeasurement target object (normal information), thereby to calculatemore accurately the correlation between the areas on the image 700 andthe areas on the illumination pattern 100. To do this, the positions onthe illumination pattern 100 corresponding to points in the blocks arecalculated. Specifically, the blocks on the image including the imageparts Pk are regarded as planes, and the distance values of the pixelsin the blocks are calculated from the shape information in the imageparts Pk on the center of the blocks. In this example, the distancevalues of the pixels in the blocks can be calculated by adding changesresulting from the normal information Aki and Ski with reference to thedistance values Dki of the image parts Pk. Next, the positions on theillumination pattern 100 are calculated based on the distance values ofthe pixels centered on the image parts Pk. Then, from the correspondencerelationship, the differences between the pixels of the blocks centeredon the image parts Pk and the pixels at the corresponding positions onthe illumination pattern. 100 are calculated. Finally, the sums ofsquares (SSDs) of the differences calculated for the pixels aredetermined, and the reciprocal numbers of the SSDs are set as thedegrees of correspondence as in the first embodiment.

The pattern acquisition unit. 210, the image acquisition unit 220, andthe candidate selection unit 260 perform the same processes as those inthe first embodiment and thus descriptions thereof are omitted.

As described above, in the embodiment, the distance values and thenormal information are treated the shape information. In addition to thedistance values, the normal information is used for calculation to allowthe shape of the measurement target object to be measured moreaccurately.

In the embodiment, the distance values and the normal information aretreated as the shape information. Alternatively, the constitutionalelements of the shape information may be any information indicative thepositions of planes of the measurement target object in athree-dimensional space (hereinafter, called position information) orany information indicative of the normals of the planes (normalinformation). The position information here may be distances,parallaxes, or three-dimensional coordinates. The normal information maybe the amount of change in the distance values relative to change in thepositions of the pixels, or three-dimensional vectors.

The shape information may be both or either of the position informationand the normal information. When some element of the shape informationis a variable, the other element may be a fixed value for use in thecalculation. For example, when the shape information is estimated withonly the position information as a variable, the normal may be assumedto be opposite to the line-of-sight direction of the imaging apparatus400 and be given as a fixed value. For example, when the shapeinformation is estimated with the normal information a variable, theposition information may indicate the general position of themeasurement target object as a prescribed value.

FIRST MODIFICATION EXAMPLE

As a first modification example, variations of a method for settingcandidates for shape information by the candidate setting unit 240 willbe explained.

The image parts Pk in which the reference shape information estimationunit 230 sets the candidates for the shape information based on theshape information may be any image parts Pk on the image 700. As in thefirst embodiment, the candidates for shape information may be set in theparts Pk within the predetermined range R surrounding the parts Fi inwhich the shape information is estimated, or the candidates for shapeinformation may be set in parts adjacent to the reference parts Fi orsome of the parts. For example, the candidates may be set for the partsadjacent to the reference parts Fi in the x-axis direction or the y-axisdirection, or the candidates may be set in the parts separated from thereference parts Fi.

When the candidate setting unit 240, the candidate evaluation unit 250,and the candidate selection unit 260 repeat the processes at S640, S650,and S660, the positions of the candidates to be set may be changed ateach repetition. For example, the candidate for shape information may beset in parts in a plus direction of the x axis of the noted referencepart in the first cycle, and the candidate for shape information may beset in parts in a minus direction of the x axis in the second cycle.

SECOND MODIFICATION EXAMPLE

As a second modification example, a method for estimating shapeinformation by the reference shape information estimation unit 230 andvariations of reference features will be explained.

A method for identifying the reference features by the reference shapeinformation estimation unit 230 may be any method to estimate shapeinformation by calculating the positons on the illumination patterncorresponding to the reference features observed on the image.

For example, when the reference features are lines, the lines may bedetected by any method for extracting the positions on the linesobserved on the image as labeled lines. The lines may be detected bylabeling edges detected by a Sobel filter or by turning the binalizedimage into fine lines with a predetermined threshold.

The detected lines may be associated by any method for calculating thepositons on the illumination pattern corresponding to the lines observedon the image. For example, as in the first embodiment, cuts may be addedat random to the lines on the illumination pattern 100 and thecorresponding positions of the lines may be calculated based on thepositions of the cuts observed on the image. In addition, referencesigns of at least one of brightness, color, and shape may be added tothe lines on the illumination pattern, and the lines may be identifiedby the reference signs observed on the image. For example, asillustrated in an illumination pattern 110 of FIG. 5, a reference signwith a different brightness value (111 in FIG. 5) may be arranged asidentification information on the lines. The reference sign may bearranged between the lines, not on the lines. Further, as illustrated inan illumination pattern 120 of FIG. 5, lines of different colors (121 inFIG. 6) may be projected as identification information, and thecorresponding positions on the illumination pattern 121 may becalculated based on the colors of the lines observed on the image.Instead of the colors of the lines, the colors of the spaces between thelines may be changed. Alternatively, the brightness value on the linesof the illumination pattern may be modulated and the correspondingpositions on the illumination pattern may be calculated based on changesin the brightness of the lines observed on the image.

The reference features may be any features that. allow the calculationof the positions on the illumination pattern corresponding to thereference features observed on the image. The reference features may belines as in the first embodiment or any other features. For example, asillustrated in an illumination pattern 130 of FIG. 5, the referencefeatures may be dots (131 in FIG. 5) different in shape from the designon the illumination pattern. In this case, the processes for detectionand association used for the reference shape information estimation unit230 to calculate the shape information on the dots 131 can be performedas described below. The dots 131 may be detected by binalizing the imageand selecting an area of a predetermined or larger size, or may bedetected by dot-shaped template matching, for example. The dots 131 maybe associated by matching with the positions on the epipolar lines ofthe illumination pattern, or may be associated based on the changedcolors and shapes of the dots. The reference features other than dotsmay have a polygonal or any other shape. FIG. 5 illuminates theillumination pattern in which the background is in white and thereference features and the design are in black. The light of the patternis reversed such that the background is projected with low brightnessand the reference features and the design are projected with highbrightness.

THIRD MODIFICATION EXAMPLE

As a third modification example, variations of design on theillumination pattern will be explained.

The elements of the design on the illumination pattern are different inat least one of shape, size, brightness, and color from the referencefeatures, and the design is not a solid design on the entireillumination pattern. Specifically, the design may be a plurality ofdots arranged at random as in the first embodiment or a plurality ofdots arranged at equal intervals as illustrated with reference sign 140of FIG. 6. Alternatively, the design may have any elements other thandots as illustrated with reference sign 150 of FIG. 6. The elements ofthe design may be the same in brightness and color, or may be differentin brightness and color among the parts of the illumination pattern.Instead of having a plurality of elements such as dots, the design maybe formed by modulating the brightness and the color tone as illustratedwith reference number 160 in FIG. 5. In any of the designs, the degreesof correspondence can be evaluated by determining the correlationbetween the parts on the image and the corresponding positions on theillumination pattern. FIG. 6 illuminates the illumination pattern inwhich the background is in white and the reference features and thedesign are in black. The light of the pattern is reversed such that thebackground is projected with low brightness and the reference featuresand the design are projected with high brightness.

FOURTH MODIFICATION EXAMPLE

As a fourth modification example, variations of a method for evaluatingthe degrees of correspondence by the candidate evaluation unit 250 willbe explained.

The method for evaluating the degrees of correspondence may be anymethod for calculating the degrees of correspondence indicating whetherthere is correspondence between the image parts and the correspondingparts with the candidates for the shape information set by the candidatesetting unit 240, based on the correlation between the areas on theimage including the image parts and the areas on the illuminationpattern including the corresponding parts. The method may be pixelmatching or block matching. In the case of block matching, thecorrelation between blocks may be represented by SSD as in the firstembodiment or SAD (Sum of Absolute Difference). The shape of the blocksmay be any shape such as square or circle. The feature amounts may beextracted from the blocks to calculate the degrees of correspondencebased on the differences in the feature amounts. The feature amounts maybe statistical values such as averages or variances of the brightnessvalues in the blocks may be feature amounts based on SIFT(Scale-Invariant Feature Transform) or the like.

In the case of block matching, as in the first embodiment, for theblocks centered on the image parts Pk, the positions Qki on theillumination pattern are calculated based on the candidates Ski for theshape information in the image parts Pk to determine the correlationbetween the areas on the image centered on the image parts Pk and theareas on the illumination pattern centered on the Qk. By this method,the positions on the illumination pattern corresponding to the centerpositions of the blocks are calculated. Alternatively, as in the secondembodiment, the positions on the illumination pattern corresponding tothe positions in the blocks may be calculated to determine thecorrelation between the image and the illumination pattern from thecorrespondence relationship.

Other Embodiments

Embodiment(s) of the present disclosure can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment (s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment (s) and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment (s) and/or controlling the one or more circuits to performthe functions of one or more of the above-described embodiment(s). Thecomputer may comprise one (yr more processors (e.g., central processingunit (CPU), micro processing unit (MPI)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

ADVANTAGEOUS EFFECTS OF THE EXAMPLES

In the first embodiment, the candidates for the shape information in theimage parts on the image are set from the shape information calculatedfrom the reference features. The solution is selected from among the setcandidates, which decreases the amount of calculation as compared to thecalculation with all the shape information within the search area ascandidates. This allows high-speed measurement of the shape informationon the measurement target object.

In the second embodiment, the shape information includes the positioninformation indicating the positions of the planes of the measurementtarget object in the three-dimensional space and the parameters for thenormal information indicating the orientations of the planes. The shapeof the measurement target object can be measured more accurately takinginto account the normal information as well as the position information.

DEFINITIONS

In the present disclosure, the illumination pattern acquired by thepattern acquisition unit may be any illumination pattern that iscomposed of a design and one or more reference features. The design isdifferent in at least. one of shape, size, brightness, and color fromthe reference features, and is not a solid design in the entireillumination pattern. For example, the design may have a plurality ofdots arranged at random or a plurality of dots arranged at regularintervals illustrated with reference sign 140 in FIG. 6. The design mayhave any shape other than dots or may be formed by modulating thebrightness and the color tone. The reference features may be any otherfeatures that. allow the calculation of the positions on theillumination pattern corresponding to the positions of the referencefeatures observed on the image in the reference shape information. Forexample, the reference features may be lines or may have any other shapesuch as circle or polygon.

The image acquired by the image acquisition unit may be anytwo-dimensional image that is captured by the imaging apparatus whilethe illumination pattern is projected onto the measurement targetobject. For example, the image may be a grayscale image or a colorimage. The image may be acquired directly from the imaging apparatus orthe image stored temporarily in the memory may be read into theapparatus.

The method for estimating the shape information may be any method forcalculating the positions on the illumination pattern corresponding tothe reference features observed on the image and estimating the shapeinformation by triangulation method. The shape information here includesat least one of the information indicating the positons of planes of themeasurement target object in the three-dimensional space and theinformation indicating the normals of the planes. The informationindicating the positons in the three-dimensional space may be distancesor parallaxes, or three-dimensional coordinates. The informationindicating the normals of the planes may be the amount of change indistance values relative to change in the positions of the pixels orthree-dimensional vectors. In addition, the shape information betweenthe reference features observed on the image may be interpolated andoutput.

The parts in which the candidates for the shape information are set bythe candidate setting unit may be any parts in the image. For example,for the shape information estimated by the reference shape informationestimation unit, the candidates for the shape information may be set inthe parts close to one another on the image or the candidates for theshape information may be set on the entire image. plurality ofcandidates for the shape information may be set in one part. In thepresent disclosure, when the candidate setting unit, the candidateevaluation unit, and the candidate selection unit repeat the processes,the parts in which the candidates are set may be changed at eachrepletion.

The method for evaluating the degrees of correspondence by the candidateevaluation unit may be any method for calculating the degrees ofcorrespondence between the candidates for the shape information set bythe candidate setting unit based on the correlation between the areas onthe image and the areas on the illumination pattern equivalent to thecandidates for the shape information. For example, the degrees ofcorrespondence may be calculated based on the SSD (Sum of SquaredDifference) or SAD (Sum of Absolute Difference) of the brightness valuesin the areas on the image and the areas on the pattern, or the featureamounts in the areas may be extracted to calculate the degrees ofcorrespondence based on the differences between the extracted featureamounts, for example. The degrees of correspondence here refer to theheights of correlation between the noted areas on the image and theareas on the illumination pattern.

The method for selecting the candidates for the shape information by thecandidate selection unit may be any method for selecting the candidatesfor the shape information with high degrees of correspondence. Forexample, the candidates for the shape information with degrees ofcorrespondence equal to or larger than the predetermined threshold valuemay be selected or the candidate for the shape information with thehighest degree of correspondence out of the candidates for the shapeinformation set for each parts may be selected.

While the present disclosure has been described with reference toexemplary embodiments, it is to be understood that the disclosure is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of priority from Japanese PatentApplication No. 2015-150502, filed Jul. 30, 2015, which is herebyincorporated by reference herein in its entirety.

What is claimed is:
 1. An information processing apparatus comprising:an acquisition unit configured to acquire an image captured by animaging apparatus while a pattern including a plurality of firstfeatures and a second feature different from the first features isprojected onto a measurement target object; an initial value acquisitionunit configured to associate the first features included in theprojected pattern and the first features included in a captured imagebased on the captured image to identify the first features included inthe captured image, and acquire an initial value of shape information onthe measurement target object in an area including the first featureswithin the captured image based on an identified result; a setting unitconfigured to set a candidate for the shape information in a first areabased on an initial value of shape information in an area surroundingthe first area within the captured image; an evaluation unit configuredto decide an area in the pattern corresponding to the first area as asecond area based on the candidate for the shape information in thefirst area, and evaluate a correlation between the first area and thesecond area based on the second feature included in the first area andthe second feature included in the second area; and a derivation unitconfigured to derive the shape information in the first area based onthe correlation evaluated by the evaluation unit.
 2. The informationprocessing apparatus according to claim 1, further comprising a decisionunit configured to decide the second area corresponding to the firstarea in the pattern based on the set candidate for the shape informationin the first area.
 3. The information processing apparatus according toclaim 1, wherein the derivation unit that derives the shape informationin the first area based on the candidate for the shape information inthe first area when the correlation between the area in the patterndecided based on the initial value of the shape information in the firstarea and the first area is higher than the correlation between thesecond area in the pattern decided based on the candidate for the shapeinformation in the first area and the first area.
 4. The informationprocessing apparatus according to claim. 1, wherein the setting unitsets the initial value of the shape information set in a reference arearesiding in a predetermined direction from the first area as thecandidate for the shape information in the first area.
 5. Theinformation processing apparatus according to claim 4, wherein thesetting unit sets a value obtained by changing the initial value of theshape information in the reference area residing in a predetermineddirection from the first area based on an amount of change between theinitial value of the shape information in the first area and the initialvalue of the shape information in the reference area, as the candidatefor the shape information in the first area.
 6. The informationprocessing apparatus according to claim 1, wherein the setting unit, theevaluation unit, and the derivation unit repeat the processes with theshape information derived by the derivation unit as the initial value.7. The information processing apparatus according to claim 1, whereinthe evaluation unit evaluates the correlation based on a differencebetween brightness in the first area within the captured image andbrightness in the second area within the pattern.
 8. The informationprocessing apparatus according to claim 1, wherein the shape informationindicates a distance to the measurement target object.
 9. Theinformation processing apparatus according to claim 1, wherein the shapeinformation includes information indicative of normals of planes of themeasurement target object.
 10. The information processing apparatusaccording to claim 1, wherein the pattern further includesidentification information for identifying the first features, and theinitial value acquisition unit associates the first features included inthe projected pattern and the first features included in the capturedimage based on the identification information included in the capturedimage to identify the first features included in the captured image. 11.The information processing apparatus according to claim 10, wherein thefirst features are lines, the identification information is arranged onor between the lines, and the second feature is arranged between thelines.
 12. The information processing apparatus according to claim 11,wherein the identification information included in the patternconstitutes cuts in the lines included in the pattern.
 13. Theinformation processing apparatus according to claim 1, wherein thesecond feature included in the pattern constitutes dots.
 14. Theinformation processing apparatus according to claim 1, furthercomprising: a projection unit that projects the pattern onto themeasurement target object; and an imaging unit that captures an image ofthe measurement target object onto which the pattern is projected. 15.An information processing method comprising: an acquisition step ofacquiring an image captured by an imaging apparatus while a patternincluding a plurality of first features and a second feature differentfrom the first features is projected onto a measurement target object;an initial value acquisition step of associating the first featuresincluded in the projected pattern and the first features included in acaptured image based on the captured image to identify the firstfeatures included in the captured image, and acquiring an initial valueof shape information on the measurement target object in an areaincluding the first features within the captured image based on anidentified result; a setting step of setting a candidate for the shapeinformation in a first area based on an initial value of shapeinformation in an area surrounding the first area within the capturedimage; an evaluation step of deciding an area in the patterncorresponding to the first area as a second area based on the candidatefor the shape information in the first area, and evaluating acorrelation between the first area and the second area based on thesecond feature included in the first area and the second featureincluded in the second area.; and a derivation step of deriving theshape information in the first area based on the correlation evaluatedby the evaluation step.
 16. A storage medium storing a programconfigured to allow a computer to serve as an information processingapparatus comprising: an acquisition unit configured to acquire an imagecaptured by an imaging apparatus while a pattern including a pluralityof first features and a second feature different. from the firstfeatures is projected onto a measurement target object; an initial valueacquisition unit configured to associate the first features included inthe projected pattern and the first features included in a capturedimage based on the captured image to identify the first featuresincluded in the captured image, and acquire an initial value of shapeinformation on the measurement target object in an area including thefirst features within the captured image based on an identified result;a setting unit configured to set a candidate for the shape informationin a first area based on an initial value of shape information in anarea surrounding the first area within the captured image; an evaluationunit configured to decide an area in the pattern corresponding to thefirst area at a second area based on the candidate for the shapeinformation in the first. area, and evaluate a correlation between thefirst area and the second area based on the second feature included inthe first area and the second feature included in the second area; and aderivation unit configured to derive the shape information in the firstarea based on the correlation evaluated by the evaluation unit.